AI621 A5 : Focal stack & Lightfield
20215584 Wonjoon Chang
Initialization
Load the light field image, and create from it a 5-dimensional array L(u, v, s, t, c).
- u, v : the coordinates on the aperture
- s, t : the coordinates on the lenslet
- c : color channels
Each lenslet covers a block of 16x16 pixels.
filename = "./data/chessboard_lightfield.png";
The original light field image:
img5d = zeros(U,V,S,T,C, 'uint8');
img5d(u,v,:,:,:) = lf((U-u+1):U:end,v:V:end,:);
Sub-aperture views
By rearranging the pixels in the light field image, we can create images that correspond to views of the scene through a pinhole placed at different poins on the camera aperture (with fixed u and v).
Create all of these sub-aperture views, and arrange them into a 2D mosaic.
- vertical dimension : increasing u values
- horizontal dimension : increasing v values
mosaic = zeros(U*S,V*T,C, 'uint8');
mosaic(s1:s2,t1:t2,:) = img5d(u,v,:,:,:);
imshow(mosaic(1:1200,1:2100,:))
imwrite(mosaic,"./result/mosaic.png");
Mosaic of sub-aperture views:
Refocusing and focal-stack generation
Refocus at different depths by combining parts of the light field.
d denotes the depth. For d=0, we obtain the image where the top of the chess board is focused.
img5d = im2double(img5d);
u_centered = (1:U) - 1 - (U-1)/2;
v_centered = (1:V) - 1 - (V-1)/2;
refocused = zeros(S,T,C);
subap = squeeze(img5d(u,v,:,:,:));
I = imtranslate(subap,[vd, ud]);
refocused = refocused + I;
refocused = refocused./num;
Refocusing results for different depths
We can easily identify that different areas are focused by shifting sub-aperture. In the next step, we will combine refocused images with d=[0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6].
In this case, I denote by 'd' the distance between the original focal plane and the synthetic focal plane. Actually, the minus should be attached if the larger 'depth' denotes the farther objects.
focal_stack = zeros(N,S,T,C);
refocused = zeros(S,T,C);
subap = squeeze(img5d(u,v,:,:,:));
I = imtranslate(subap,[vd, ud]);
refocused = refocused + I;
focal_stack(i,:,:,:) = refocused./num;
filename = "./result/refocus-"+i+".png";
imwrite(squeeze(focal_stack(i,:,:,:)), filename);
All-focus image and Depth from defocus
To merge the focal stack into a single all-focus image, we first determine per-pixel and per-image weights. Here is the detail procedure:
img = squeeze(focal_stack(i,:,:,:));
low = imgaussfilt(lumi, sig1);
sharp = imgaussfilt(high.^2, sig2);
Once we obtain the sharpness weights, we can compute the all-focus image as:
img = squeeze(focal_stack(i,:,:,:));
allfocus(:,:,c) = allfocus(:,:,c) + squeeze(W(i,:,:)).*img(:,:,c);
W_sum = W_sum + squeeze(W(i,:,:));
allfocus(:,:,c) = allfocus(:,:,c)./W_sum;
In addition, we can create a per-pixel depth map:
depth = depth + squeeze(W(i,:,:))*d;
minD = min(depth,[],"all");
maxD = max(depth,[],"all");
depth = (minD - depth)/(maxD - minD) + 1;
Combining results
I think that the first case (sigma1=5, sigma2=5) is the best. With larger sigma1, the high frequency component may not be emphasized enough and then the result becomes somewhat blurry. With larger sigma2, the depth map becomes too blurry to recognize the objects. With very small sigma1, sigma2, the image and the depth map can have noisy components.
My own images
focals = zeros(4,640,640,3);
filename = "./data/focal"+i+".jpeg";
focals(i,:,:,:) = imread(filename);
img = squeeze(focals(i,:,:,:));
low = imgaussfilt(lumi, sig1);
sharp = imgaussfilt(high.^2, sig2);
allfocus = zeros(640,640,3);
img = squeeze(focals(i,:,:,:));
allfocus(:,:,c) = allfocus(:,:,c) + squeeze(W(i,:,:)).*img(:,:,c);
W_sum = W_sum + squeeze(W(i,:,:));
allfocus(:,:,c) = allfocus(:,:,c)./W_sum;
Original images:
In the combined image, we can see that all objects are clearly visible.
maxI = max(allfocus, [], "all");